Towards maturity of information security maturity criteria: six lessons learned from software maturity criteria

Traditionally , information security management standards listing generic means of protection have received a lot of attention in the field of information security management. In the background a few information security management-oriente d maturity criteria have been laid down. These criteria can be regarded as the latest promising innovations on the information security checklist standard family tree. Whereas information security maturity criteria have so far received inadequate attention in information security circles, software maturity endeavours have been the focus of constructiv e debate in software engineering circles. Aims to analyze what the alternative maturity criteria for developing secure information systems (IS) and software can learn from these debates on software engineering maturity criteria. First, advances a framework synthesized from the information systems (IS) and software engineering literatures, including six lessons that information security maturity criteria can learn from. Second, pores over the existing information security maturity criteria in the light of this framework. Third, presents, on the basis of results of this analysis, implications for practice and research. Rifkin, 2001; Voas, 1999). The existing critiques of software engineering maturity models provide good lessons on the problems of maturity endeavours: why should we not learn from our fellow scholars and practitioners in the fields of software engineering and IS, and thus avoid repeating the same mistakes again? Thus the aim of this paper is to scrutinize the alternative information security maturity approaches using these lenses. By unveiling the weaknesses of existing information security maturity criteria this paper has particular relevance for people involved in developing such criteria. In addition, this study also has relevance for ordinary security practitioners who are applying such criteria. For such practitioners, this study reveals some crucial weaknesses of existing information security maturity criteria. We hope that by being aware of these potential pitfalls, practitioners may be better prepared to deal with possible problems originating from them. A preliminary outline of this study appeared in the Proceedings of the 17th International Conference on Information Security, Cairo, Egypt, May 6-8, 2002. The rest of the paper is composed as follows. The second section provides an overview of the alternative maturity approaches, points out their links to related information security management standards and presents the framework for this study. The third section analyses the alternative information security management-oriented maturity standards. The fourth section discusses the findings of the paper while also proposing future research directions and questions which maturity standards should be met. The fifth section concludes by summarizing the key contributions of the paper. Research design, an overview of the alternative maturity approaches and the framework for analysis Research design The overall research approach taken in this paper is that of conceptual analysis. The hermeneutic circle is also utilized in this paper. The hermeneutic circle is an interpretive (see Klein, and Myers, 1999, 2001; Walsham, 1996) and a conceptual-analytical research method. It is commonly used by historians, philosophers and theologians to reveal in a document something that is not explicitly present in it (Mautner, 1996, p. 188). Because this study is only about discovering the assumptions of different information security management approaches that their authors have not explicitly stated, the hermeneutic circle is a natural research methodological choice. In order to `̀ validate’’ our findings, i.e. to show how we came to a particular conclusion, we have included relevant citations from the original material. Research criteria and an overview of the alternative IS security maturity approaches In selecting the information security maturity criteria to be analysed, the following criteria have been adopted. First, the maturity criterion must be a ready package, i.e. it must offer practical prescriptions for evaluating maturity. Hence, possible meta-analyses and pure research agenda papers on information security maturity criteria would not meet this criterion and are therefore outside the scope of this paper. Second, a criterion to be included in this analysis needs to be an information security management-oriented maturity criterion, not a computer security one[1]. Five information security maturity approaches meet the first criterion, including adaptive software security metrics (Voas et al., 1996), the common criterion (e.g. Caplan and Sanders, 1999; Chokhani, 1992), the information security program maturity grid (Stacey, 1996), software security metrics (Murine and Carpenter, 1984), the systems security engineering-capability maturity model (SSE-CMM, 1998a, b). An approach labelled as `̀ adding security concerns into maturity models’’ by Krzanik and SimilaÈ (1994) does not meet these criteria. This approach is particularly influenced by the Bootstrap SW maturity criterion (Kuvaja et al., 1994). This approach differs from the three others in the sense that the approach of Krzanik and SimilaÈ is more an abstract issue paper than a concrete method to be used as such. Hence, it does not meet the first selection criterion, and we are obliged to leave it out of the scope of the present study. Only three maturity approaches seem to meet the second selection criterion: the information security program maturity grid (Stacey, 1996), software security metrics (Murine and Carpenter, 1984), and the systems security engineering-capability maturity model (SSE-CMM, 1998a, b). The approaches under the banner of `̀ evaluation standards’’, the common criterion being perhaps the most notable (Abrams and Podell, 1995; Caplan and Sanders, 1999; Chokhani, 1992), do not meet the second selection criterion. These [ 211 ] Mikko Siponen Towards maturity of information security maturity criteria: six lessons learned from software maturity criteria Information Management & Computer Security 10/5 [2002] 210±224 evaluation standards are focused on technical aspects, computer systems and/or the very end of the development process or software products (see Overbeek, 1995) with the result that they cannot be regarded as information security management-oriented maturity endeavours (Baskerville, 1992). For these reasons, such technical or computeroriented standards are omitted from the present analysis. For the same reason, an assessment approach called adaptive software security metrics, proposed by Voas et al. (1996), was seen as outside of the scope of this paper. The adaptive software security metrics approach is targeted at the very end of the development process, being at the coding level for fixing bugs. Such an approach is more relevant to assessing computer systems ± or software particularly ± than ISs that encompass socialorganizational dimensions. Table I summarizes the information security management-oriented maturity approaches. These approaches (Table I) are considered next. SSE-CMM is a cognate of the capability maturity model (CMM) and ISO SPICE, both (CMM, SPICE) used to determine and improve the maturity of software processes with the help of five maturity levels (see Herbsleb et al., 1997; Paulk et al., 1993; Shere and Versel, 1994). The first version of SSECMM appeared in 1994 and the current, second version, in 1998 (SSE-CMM, 1998a, b), which is the version considered in this analysis. SSE-CMM has received attention only among North American scholars and practitioners (Ferraiolo and Sachs, 1996; Hefner, 1997; Hopkinson, 2001; SSE-CMM, 1998a, b). SSE-CMM, like CMM, put forward five maturity stages where the first stage denotes the lowest level of maturity and the fifth stage the highest stage of maturity. Each stage consists of a fixed number of security processes (a sequence of steps performed to achieve a certain objective), i.e. security activities which the organization should meet. To determine the maturity level of an organization, these security processes are trawled through in the form of a questionnaire consisting of `̀ yes/no/don’t know’’ questions (e.g. `̀ order risks by priority’’ ± `̀ yes/no/don’t know’’) within 22 process areas (such as PA03: assess security risk; PA12: ensure quality). Yet, SSM-CMM includes the concept of exploratory (freeform) questions to resolve possible inconsistencies and unsupported answers (no evidence) from questionnaires. With the help of SSM-CMM, organizations can improve the maturity of their IS security through incremental improvements (Ferraiolo and Sachs, 1996). SSE-CMM consists of two parts, a model (consisting of five maturity levels, as mentioned) and an appraisal method. The latter is a four-phased (planning, preparation, on-site, reporting) method with which to conduct the evaluation of organizations’ maturity on the basis of SSM-CMM model. Stacey’s information security program maturity grid Stacey’s (1996) information security program maturity grid also stems from the CMM and particularly Crosby’s (1979) approach labeled as `̀ the quality management maturity grid’’. As in the case of SSE-CMM, Stacey (1996) proposes five stages in order of increased maturity: 1 uncertainty (a total lack of understanding of information security ± security is a hindrance to productivity); 2 awakening (realization of the value of security ± but inability to provide resources and money for security); 3 enlightenment (security is a must ± as well as resources and money for security ± organizations need also to prevent violation, instead of merely recovering from incidents); 4 wisdom (security development reflects organizations’ environmental factors and needs ± all users are empowered in terms of information security); and Table I The alternative maturity approaches Name Key ideas Sources of influence Key advocates/references SSE-CMM Five maturity levels CMM SSE-CMM (1998a, b); Ferraiolo and Sachs (1996); Hefner (1997); Hopkinson (2001); Annual ISSEA conference Information security program maturity grid Five maturity levels CMM, the quality management maturity grid Stacey (1996) Software security metrics 11 high-level security criteria and five milestones Murine and Carpenter (1984) [ 212] Mikko Siponen Towards maturity of information security maturity criteria: six lessons learned from software maturity criteria Information Management & Computer Security 10/5 [2002] 210±224 5 benevolence (continuous security process improvement through research and practice). Also, like SSE-CMM, Stacey’s information security program maturity grid incorporates several more detailed prescriptions associated with each maturity level. Murine-Carpenter maturity criterion The incentive behind the maturity criterion of Murine and Carpenter (1984) lies in building quantifiable metrics for the information security maturity of systems and software (Murine and Carpenter, 1984 p. 208). They feel that software quality metrics do not address the security aspect adequately enough, resulting in the need to elaborate software security metrics. They list 11 highlevel security criteria (Murine and Carpenter, 1984 p. 212) and five milestones (Murine and Carpenter, 1984, p. 213), i.e. phases in software development where the maturity of security in software development should be analyzed in the light of their maturity criteria. A framework for analysis Maturity endeavours started in the field of software engineering. The US Department of Defense (DoD) misjudged the ability of several its contractors to develop mature SW with the result that the DoD deemed it imperative to mobilize a maturity standard (Bollinger and McGowan, 1991; O’Connell and Saidian, 2000 p. 28). Accordingly, several SW maturity criteria have been presented (Kuvaja et al., 1994). Yet, several criticisms have been levelled against the SW maturity criteria. These problems about the existing software maturity models form the analytical framework of this study, i.e. six lessons from which information security maturity criteria can learn. These perceived problems are summarized in Table II. The problems summarized in Table II will now be discussed in more detail. Do the criteria entail conventionalism or an operational focus? Rifkin (2001) recognized that SW maturity criteria focus on operational issues alone. In fact, the SW maturity criteria do not support innovation at all, since they do not encourage organizations to innovate, but instead insist on the use of existing and well-known methods and practices. This is known as conventionalism. Rifkin (2001) therefore reported that organizations regarding innovation as their main competitive strategy gain nothing from the adaptation of SW maturity criteria. Furthermore, organizations using very state-of-the-art security solutions, perhaps even participating in the creation of new security contributions (e.g. the highest level of information security program maturity grid by Stacey), may not rank high in maturity estimations, as the new solutions are not recognized by the maturity criteria (as the criteria are based on old information). The point of the operational focus is to ponder whether the information security maturity criteria uphold conventionalism (and have an operational focus), or do these criteria support reforms and innovations? Are the criteria naturalistic-mechanistic? The naturalistic-mechanistic view refers to the idea that phenomena can be quantified and controlled. Positivism, which claims that the methods of natural sciences should form the basis of all sciences, is the most wellknown example of this view (see Hirschheim, 1985; Ray, 2000). Also, so-called behaviorists in the field of the behavioral sciences advocated a similar view under the direction of B.F. Skinner, although with inferior results. When it comes down to software development, Humphrey (1988) states several hints giving the impression that he is a proponent of the naturalistic-mechanistic view. To provide examples, Humphrey sees that the software process should be `̀ predictable’’ (Humphrey, 1988, p. 73), Table II The recognized problems of existing SW maturity approaches Problems Problem description Operational focus Conventionalism vs emphasis on operational issues, does not support innovation Naturalistic-mechanistic Naturalistic-mechanistic (positivistic) world-view Stable, non-emergent, organization structures and functions Do the maturity standards support IS/SW development in emergent organizations? Double standard Distort the truth Spot focus The focus of inspection is on predefined spots ± not a holistic posture of overall maturity The degree of ambiguity Reference-only, subjective, partially objective and objective [ 213 ] Mikko Siponen Towards maturity of information security maturity criteria: six lessons learned from software maturity criteria Information Management & Computer Security 10/5 [2002] 210±224 predefined, repeatable (Humphrey, 1988, p. 74) and stable (Humphrey, 1988, p. 75). The aforementioned objectives, namely predefined, repeatable and predictable, accord well with this scientific posture, as it leans on the paradigm of positivism. It is, however, erroneous to contend that such a view of natural science is valid in all areas of science (i.e. positivism). The fallacy lies in the fact that the naturalistic-mechanistic view may be valid in the arena of the natural sciences. However, this is not the case in the social or humanistic sciences, including IS[2]. At least currently, human behavior cannot be fully deduced from specific causal reactions (Abrahamsson, 2001). If someone were able to this, there would most likely be no need for information security, as we would be able to manipulate or induce the necessary causal reactions, with the result that people would not commit security violations. Advocates of software maturity models may try to avoid this objection by saying that the software can be developed within the research paradigm of natural sciences/ positivism. The next quotation illustrates this view. Humphrey (1988, p. 74) argues that: While there are important differences, these concepts [maturity, statistical process control] are just as applicable to software as they are to automobiles, cameras, wristwatches, and steel. A softwaredevelopment process that is under statistical control will produce the desired results within anticipated limits of cost, schedule and quality. Such a viewpoint entails problems, however. Voas (1999, p. 120) put forward a rebuttal to Humphrey’s claim, pointing out that software development is an inventive process; it is not a manufacturing process (as assumed in the citation from Humphrey). Yet, Pfleeger (1999, p. 33) has pointed out that software maturity criteria have relied upon the natural science (see Harre , 2000) conception of science: We [software engineers] seek relationships to help us understand what makes good software. We then apply what we learn so that we get good software more often. Our search is based in large part on the notion of cause and effect. Pfleeger further remarks that this idea of `̀ cause and effect’’ is fallacious because the processes are not natural, but human made, i.e. social processes. Nevertheless, even if one still insists that software can be developed on the basis of the natural science paradigm, the problem with respect to security maturity criteria remains that these models ± at least SSE-CMM and information security program maturity grid ± are applied to secure systems, not just secure software development. Hence, even if the naturalistic-mechanistic view would be adequate for software development, it is definitely not an adequate framework for approaches aimed at securing organizations’ ISs. Do the maturity standards support IS/SW development in emergent organizations? SW maturity standards seem to imply that the IS/software development takes place in a stable environment, for two reasons. First, current SW maturity models rely on strict waterfall models (Boehm, 2000). Second, they stem from the analogy cited from Humphrey between software development and manufacturing, which, however, is argued here to be a mistake. This analogy is flawed since traditional manufacturing is largely a stable and fine-tuning replication process, whereas software development is more a creative design process (Bollinger and McGowan, 1991, p. 36). It has been argued that the same goes for IS development. For example, increasing numbers of organizations are reported to be emergent (i.e. turbulent), as opposed to stable, in terms of their business environment, and hence they require appropriate means for developing SW/IS as the current IS/SW development methods are suited to stable organizations (Baskerville et al., 2001; Baskerville and Pries-Heje, 2001; Truex et al., 1999, 2000). Their basic claim is that any modern IS/SW development method should support the requirements posed by emergent organizations. In other words, a successful method should be able to adapt rapidly to ever-changing requirements owing to a fastpaced business environment. This issue is also relevant for information security maturity criteria. Given that the business environment is a turbulent one requiring that IS/software be developed rapidly, there may not be time to wait for bureaucratic and long-term security processes to take place. Can maturity criteria tackle the problem of the double standard? Companies may be under pressure to perform well in terms of maturity as a good maturity rating results in good publicity and an increase in business competence (O’Connell and Saidian, 2000, pp. 32-3). The problem of double standard refers to a situation where an organization manipulates its results in order to look better in a maturity evaluation. O’Connell and Saidian (2000, pp. 33-4) describes several such tricks that an organization may play. Moreover, Bollinger and McGowan reported how [ 214] Mikko Siponen Towards maturity of information security maturity criteria: six lessons learned from software maturity criteria Information Management & Computer Security 10/5 [2002] 210±224 maturity evaluators schooled by DoD are trained to `̀ distinguish genuine answers from attempts at obfuscation or even outright falsification’’ (Bollinger and McGowan, 1991, p. 28). They also provide a few countermeasures for this problem, including conducting online empirical evaluations (not just paper-based), requiring two sources of confirmation, choosing a representative evaluation team. However, is it clear that these proposed cures do not remove these problems. The problem of the double standard is at least as relevant an issue in the field of information security as in the field of software engineering. One key objective of having the concept of maturity levels is to sketch `̀ objective and universal’’ criteria, which are able to indicate the maturity of all kinds of organizations. By developing such criteria, the major objective is not to show the maturity level for the organization adopting the maturity criteria (interorganizational) ± although this is also a secondary objective of maturity standards ± in which case the problem of the double standard is not so crucial. As with CMM, one crucial aim of setting up information security maturity criteria is to assist other organizations, third parties, business partners, etc. to ensure that organizations which they deal with have certain level of information security. Recognizing this, it is justified to presume that the increasing recognition and use of a widely accepted maturity criteria would increase the incentives of an organization to score high in terms of these IS security maturity criteria (see O’Connell and Saidian, 2000). Spot focus Bollinger and McGowan (1991, p. 33) levelled the criticism that the focus of maturity inspection is on prefixed spots, the result being that the criteria do not pay any attention to a holistic overall maturity posture. To illustrate this problem, Bollinger and McGowan submit the case where one person painting, say, a car, does the job very well in certain spots, leaving other places without any paint (case A), scores equally high on the maturity scale as another person painting the whole car consistently well (case B); hence, the label spot focus. Clearly, the latter option (B) is more mature in terms of painting, but as the SW maturity criteria only examine certain spots, they lack the overall maturity estimation, and both A and B would scale equally high on the maturity scale, given that the measurement idea identical to software maturity criteria is used (Bollinger and McGowan, 1991). Clearly, the problem of spot focus is relevant to the assessment information security maturity criteria. If such faults were a characteristic of security maturity criteria, it would result in organizations securing ISs/ software possibly concentrating heavily only on certain places in order to increase their maturity level, whilst bestowing less or no attention on certain other aspects not covered by the maturity model (as in the case A in the aforementioned example). The aim of this viewpoint is to explore whether the information security maturity criteria succumb to spot focus fallacy? What degree of ambiguity is present in maturity criteria? Four categories of degrees of ambiguity are proposed (Pfleeger et al., 1994): referenceonly, subjective, partially objective and objective. Reference-only means a situation where the standard prescribes something which does not provide any concrete means for ensuring compliance. For example, a prescription: `̀ risk analysis should be carried out’’ without any indication of what the results of good risk analysis would be is an example of a reference-only practice. The fallacy is the lack of information concerning, for example, what good risk analysis practice should include and what is the scope of such risk analysis. The subjective situation is the case when the goodness/scope of a practice is left to be determined on the basis of an expert judgment. For example, a prescription `̀ risk analysis method needs to be applied with respect to relevant assets’’ is a case in point. Partially refers to a situation which is more concrete than that of subjective practice, but still entailing some ambiguous information (e.g. `̀ risk analysis needs to be accomplished with respect to relevant assets included’’). Objective denotes prescriptions which give explicit guidelines with minimum inarticulateness. The advantage of the objective practice is the exactness it has compared to other alternatives. The weakness is that such general standards, as the objective one exemplifies, is difficult to build ± and not wise, at least in all cases (as organizations, their business and security requirements/needs vary). The referenceonly alternative is ranked as the worst alternative owing to the several important open questions it does not address. Analysis of the information security management-oriented maturity approaches The assumptions underlying the various maturity criteria will be analyzed next. [ 215 ] Mikko Siponen Towards maturity of information security maturity criteria: six lessons learned from software maturity criteria Information Management & Computer Security 10/5 [2002] 210±224 Operational focus SSE-CMM SSE-CMM does not pay close attention to this problem. At any rate, it does not trumpet the idea that evaluators should seek innovations. However, it does leave evaluators with some freedom to tailor some parts of process areas using their professional judgement. Yet, SSECMM says that one `̀ may tailor some aspects . . . to satisfy particularly needs’’ (SSE-CMM, 1998b, p. 71). Unfortunately, it is not clear what this means exactly: what are some aspects? Or can one modify everything? Information security program maturity grid At the earlier level, the focus of this criterion is operational, as the following two examples from the second maturity stage illustrate. First: `̀ End-users view security restrictions as an unnecessary hindrance’’ (Stacey, 1996). Second: `̀ The end-users’ productivity is affected now both by the security incidents and by the safeguards set in place to protect the system’’ (Stacey, 1996). These examples indicate that organizations at the second stage do not stimulate security innovation; on the contrary, their security practice seems to debar normal practices. However, the fifth stage seems to pave the way to innovations. Stacey (1996) requires that in order to qualify to the fifth stage, security people in organizations need to participate in research projects and `̀ its security professionals will be likely to achieve notoriety through presentation at . . . conferences, . . . journals . . .’’. We see this citation clearly implying that the fifth (the highest) stage imposes requirements for security innovations to be created by research/development processes selected results, which are reported to science through conferences and journals. Murine-Carpenter maturity criterion On the one hand, this criterion does not support the idea of innovations for two reasons. First, it does not give direct hints in favour of innovations (as Stacey’s criterion does). Second, the milestones are based on an existing typical software development lifecycle including the stages `̀ system security requirements’’, `̀ software system security requirements’’, `̀ functional security architecture’’, `̀ modular security gating’’ and `̀ security testing’’ (Murine and Carpenter, 1984, p. 213). On the other hand, the MurineCarpenter maturity criterion does not rule out the possibility of innovation. In fact, the use of innovations may become reality as this criterion gives a lot of freedom for developers to choose particular techniques/methods within each milestone (Murine and Carpenter, 1984, p. 213). However, the fact that the milestones are a must for all organizations hinder the use of fundamental innovations that are not in synch with the milestone concept. Naturalistic-mechanistic assumptions SSE-CMM seems to contain ideas similar to those of behaviourism. For example, if A, then B, is an example of such a naturalisticmechanistic causal law. SSE-CMM exhibits this idea, as the following citation illustrates: `̀ The SSE-CMM was developed with the anticipation that applying the concepts of statistical process control to security engineering will promote the development of secure systems and trusted products within anticipated limits of cost, schedule, and quality’’ (SSE-CMM, 1998a). As seen in this citation, SSE-CMM takes Humphrey’s analogy of manufacturing and SW development, along with the need of statistical control, seriously (see Ferraiolo and Sachs, 1996). The next citation supports this interpretation: `̀ Process capability is defined as the quantifiable range of expected results that can be achieved by following a process’’ (SSE-CMM, 1998a, p. 22). Also the fact that the fifth (highest) maturity level is aimed at `̀ establishing quantitative goals . . .’’ (SSE-CMM, 1998b, D.4) indicates the role of the naturalistic-mechanistic view as it bears on SSE-CMM. Information security program maturity grid We do not find any naturalistic-mechanistic assumptions underlying the information security program maturity grid. This criterion is not founded on observing industrial practices whilst trying to recognise cause-effect relations. Yet, this criterion does not state explicitly that the behaviour of users can be manipulated by cause-effect manner (such as whenever A, then B where, for example, A denotes `̀ awareness program accomplished’’ and B `̀ users are motivated’’). However, it uses an expression which one might regard as imparting the flavour of a naturalisticmechanistic assumption: `̀ because of the thorough security awareness training program, end-users are more vigilant and tend to initiate more incident reports’’ (Stacey, 1996). Even though Stacey in the citation above provides an ideal and simplistic picture of hoped results of awareness programs, we do not find his thinking couched in a naturalisticmechanistic view. Murine-Carpenter maturity criterion SSM provides many hints of a naturalisticmechanistic view. On the one hand, it repeatedly clearly states the aim of laying [ 216] Mikko Siponen Towards maturity of information security maturity criteria: six lessons learned from software maturity criteria Information Management & Computer Security 10/5 [2002] 210±224 down a quantifiable maturity criterion (Murine and Carpenter, 1984, pp. 207-8). On the other hand, it does not directly assert the existence of certain mechanistic-causal relationships. However, the fact that SSM is concentrated wholly on technical aspects suggests that its worldview is very much of a naturalistic-mechanistic kind. The fact that the human component is not recognized or considered at all by SSM further supports our conclusion. Assumption on stable, non-emergent, organization structures and functions None of these maturity standards fully recognizes the issues which secure SW/IS development in emergent organizations pose. Some of the standards do better than others, and we shall analyze each criterion in detail next. SSE-CMM SSE-CMM adopts highly stable IS security development approach. In fact, the way IS security development is contrived, pursuant to SSE-CMM, makes it perhaps the most rigid of the alternative maturity approaches. The whole maturity approach itself is close to 1,000 pages long, the evaluation process is formal (and long), proceeding through all points and stages starting from the first stage. Moreover, the formulation and functioning of the appraisal process is formal and includes bureaucratic elements. For example, there are several appraisal/ appraised organizational roles that are recognized to be fulfilled in the evaluation process, such as appraisal facilitators, evidence custodian, voting members, observers (mentioned in appraisal organizations) and site coordinator, executives, executive spokesman, project leader and practitioner (mentioned in roles in the appraised organization (SSE-CMM, 1998b)). Such bureaucracy in evaluating the security level of an organization may be too much, particularly for small emergent organizations. Yet, the appraisal process includes 18 phases and sub-phases; planning (three sub-phases), preparation (four subphases), on-site (seven sub-phases), and reporting (four sub-phases). Things could be much worse from the point of view of emergent organizations. Fortunately, SSECMM gives the evaluators/target of evaluation freedom to choose the particular goals of the evaluations. Moreover, the assessment tool includes the concept of `̀ tailorable parameters’’ allowing the evaluators e.g. to decide autonomously the scope of the evaluation with respect to these parameters. Information security program maturity grid Organizations scaling at the low level in terms of this maturity criterion are those not yet ready for emergent IS development. A few examples follow: `̀ Security is viewed as a commodity that can be bought on the open market’’. This example from the second maturity level indicates an assumption that general security solutions can be unearthed from standards, for example. Here is another example from the second level: `̀ The officer will identify the significant threats and develop policies and procedures in response to the most frequently occurring crises’’. This implies that a stable environment is assumed for organizations belonging to the second level ± they are waiting for `̀ most frequently occurring crises’’. Yet, Stacey (1996) also recognizes the potential complications if organizations follow the pattern of stable organizations: `̀ Losses may be high, especially when they do not follow the historical trend’’. In the third level, we see an increasing recognition of the idea of emergent organizations, as the following passage from the third level shows: `̀ security is no longer viewed solely as a commodity that can be purchased. Rather, information security must be designed consistent with an enterprise’s needs ± it must be designed from within.’’ This is a first step towards the recognition that in an emergent organization security needs to originate from the organization’s (unique) mission, not from outside in the form of a generic security package. To give a second example: `̀ Previously prepared risk analysis becomes stale and demonstrates loose applicability to the evolving environment’’. The recognition of the inadequacy of the previous risk analysis in evolving an environment in the latter citation also demonstrates a shift in thinking towards the idea of IS/SW development in emergent organizations. The fourth level continues by giving support for the requirements posed by an evolving business environment. Two examples from the fourth maturity level follow. First, in order for an organization to be at the fourth level, it should: `̀ closely reflect the enterprise’s environment and respond to the enterprise’s evolving needs’’. As a second example: `̀ Threats are continually reevaluated based on the changing threat population and on security incidents’’. Both these examples clearly demonstrate a readiness on the part of fourth stage organizations in an era of emergent organizations. Finally, we see that the fifth stage supports the idea of emergent organizations as well. Consider the following [ 217 ] Mikko Siponen Towards maturity of information security maturity criteria: six lessons learned from software maturity criteria Information Management & Computer Security 10/5 [2002] 210±224 citation: `̀ continual information security process improvements through research and participation and the sharing of knowledge in public and professional forums’’. This is an example of an ongoing analysis of a natural situation for emergent organizations (see Truex et al., 1999). On the negative side, Stacey’s approach does not pay attention to the requirements arising from a fast pace of development in the case of emergent organizations. Murine-Carpenter maturity criterion Regarding the question of stable versus emergent organizations, the MurineCarpenter maturity criterion lies somewhat between these two views. One the one hand, it suggests well-known aspects such as the five milestones. On the other hand, its prescriptions, such as milestones, lie at a very high level of abstraction and leave the choice of particular techniques/methods for the practitioners or users of the maturity criterion (Murine and Carpenter, 1984, p. 213). Yet, purely going through this criterion does not necessarily require a huge number of manpower/working hours, as opposed to SSE-CMM, and generally it is not very rigid (also as opposed to SSE-CMM). Our conclusion is that the Murine-Carpenter maturity criterion may be of use to organizations which purely wish to improve their software security in an emergent environment. Double standard As we understand it, no maturity standards explicitly address this issue. The information security program maturity grid and the Murine-Carpenter maturity criterion do not come close to this issue at all. SSE-CMM comes closest to this issue, and therefore is the only one considered (see below). SSE-CMM The closest recognition of double standard is the following statement: `̀ Evidence must be weighed by the appraisal team in that the source of the information is taken into consideration when the evidence is considered. For example, evidence from questionnaires or interviews may be considered less `valid’ in that they are more prone to misuse or misunderstanding. Thus, the team might look for a certain amount of corroborating evidence from other sources, such as documents’’ (SSE-CMM, 1998b, p. 80). Two things emerge from this citation. First of all, SSM-CMM does not provide any explicit guidance on recognizing and addressing the problem of the double standard, excepting evaluators’ intuitions. Second, we understand this citation to imply that the evaluators are advised to pay more attention to documentation than interviews (consider: `̀ may be considered less valid’’). However, the existence of documentation per se cannot be regarded as meeting the case, since documentation is also easy to fabricate and tamper with in an era of computers. It is also easy to re-cycle documents without doing the actual things described in the documents. Spot focus SSE-CMM In case of SSM-CMM, on the one hand the process areas (and particular detailed questions within each process area) are the generic and predefined spots: `̀ while the PAs are generic, the sponsor may tailor some aspects of PAs to satisfy particular needs’’ (SSE-CMM, 1998b, p. 71). In that light, SSMCMM succumbs to the fallacy of the spot focus. On the other hand, the organization can select the process areas to be included in the study. Yet, with respect to each process area there are a few `̀ tailorable parameters’’ which organizations may modify. However, these two points do not remove the spot focus fallacy; the process areas and tailorable parameters are prefixed, with the result that one cannot make one’s own maturity spots. SSE-CMM also includes a hint, which might be interpreted as allowing refinement: `̀ although the SSAM [the assessment method of SSE-CMM] is a defined method, . . . organizations may need to further refine particular aspects of the method to meet individual sponsor goals and expectations. All refinements must be documented and agreed upon by the sponsors and the appraisal organization.’’ However, it is not clear whether this is constrained by tailorable parameters, or whether one can modify anything as long as the modifications are agreed upon. Information security program maturity grid In the fifth stage, the objectives are defined loosely with a result that Stacey’s information security program maturity grid eschews the fallacy of the spot focus. Murine-Carpenter maturity criterion The Murine-Carpenter maturity criterion entails the spot focus fallacy. The five milestones and 11 security criteria are universal, i.e. they should be embedded in every mature secure software development endeavor. However, the fact that milestones are only mandatory at a very abstract level whilst concrete methods/techniques to be used at low levels are non-mandatory may decrease the problem of the spot focus. Degrees of ambiguity The degrees of ambiguity with respect to SSE-CMM range from reference only to [ 218] Mikko Siponen Towards maturity of information security maturity criteria: six lessons learned from software maturity criteria Information Management & Computer Security 10/5 [2002] 210±224 objective. To some extent, degrees of ambiguity such as the subjective will suffice, and may even be necessary. Reference-only ambiguity is blameworthy, however. SSECMM resorts frequently to the use of reference-only practice. To give an example, we see the following point in Base Practice 1: `̀ manage security awareness, training, and education programs for all users and administrators’’ (SSE-CMM, 1998b, D.2), as an example of reference-only practice. SSECMM does not really indicate (not in the SSECMM base practice explanation part) what such concepts as `̀ awareness’’ or `̀ management’’ mean. Does the term awareness refer to `̀ being aware of something [security]’’, or to a state of affairs where employees are fully committed to security policy ± the latter is not stated by SSE-CMM. It results from the reference-only practice that SSE-CMM does not provide any guidance on how one can know that employees are aware of ± or committed to ± security guidelines. Information security program maturity grid The degree of ambiguity of Stacey’s (1996) information security program maturity approach entails both reference-only and subjective views. Imperatives such as organize a `̀ thorough information security training program for end-users’’ are an example of a reference-only view. It does not imply what a good information security training program includes. We also found a subjective pattern `̀ users are empowered and encouraged to evaluate and develop their own risk-based management strategies and to customize the enterprise’s existing information security program to respond to their own needs’’. This example provides clear guidance with respect to customization, given that end-users know their own needs, but it does not give any indications regarding the process of `̀ risk-based management strategies’’, such as what a good process for accomplishing such risk-based management strategy might contain. Murine-Carpenter maturity criterion The Murine-Carpenter maturity criterion incorporates all degrees of ambiguity. On the negative side, the criterion utilizes a great deal of reference-only practice, therefore offering no practical help to developers. Sometimes the objective practice may be fallacious, as well, as indicated in the second section. For example, consider the fifth security-testing milestone: `̀ System is tested for an illegal entry at all levels’’ (Murine and Carpenter, 1984, p. 213). This is an objective practice, at least insofar as the term `̀ illegal’’ is concerned. However, security people may not want to prevent only `̀ illegal’’ entry into a system, but rather unauthorized or unwanted entry. Discussion, limitations and implications of the findings In spite of the fact that information security literature abounds in discussion of standards, the existing maturity standards/ criteria for securing IS/software have largely been ignored. This paper analysed the existing maturity approaches for securing IS/software. In addition to self-assessment, the maturity approaches are aimed at demonstrating to organizations and the public/third parties/customers confidence in the security level/maturity of an organization. In fact, it is the latter factor which separates general information security management standards (mainly for inter-organizational self-assessment) from information security maturity managementoriented endeavours (inter-organizational self-assessment and public dimension assessment). Limitations of the study The research approach of this study was conceptual analysis (see JaÈrvinen, 1997, 2000) combined with a hermeneutic research method favored by historians, philosophers and theologists to interpret texts. Hermeneutics is of Greek origin and can be defined as `̀ the art of finding something from the text that is not there’’ (Mautner, 1996, p. 188). As the interpretation of texts constitutes the essence of this study, hermeneutic analysis is a natural choice of approach. However, this means that the results are based on our interpretations (Gadamer, 1989). This is the source of the main limitations of this study. To deal with this issue, we have adduced citations where possible to indicate the origins and validity of our interpretations (see Iivari et al., 1998). Moreover, the results of this study should not be interpreted to mean that if an information security maturity criterion fails to tackle all six problems, then the criterion is consequently invalid. Instead, the application of a criterion needs to be pondered on a case-by-case basis bearing these weaknesses in mind. Finally, the problems presented in this study are not an exhaustive list of all the possible problems residing in information security maturity criteria. Discussion of the key results The results are resummarized in Table III. [ 219 ] Mikko Siponen Towards maturity of information security maturity criteria: six lessons learned from software maturity criteria Information Management & Computer Security 10/5 [2002] 210±224